Solid-state image pickup apparatus, correction method, and electronic apparatus

ABSTRACT

The present disclosure relates to a solid-state image pickup apparatus, a correction method, and an electronic apparatus, enabled to suppress an apparent uncomfortable feeling of an image output from a solid-state image pickup apparatus in which pixels of different OCL shapes are mounted mixedly. A solid-state image pickup apparatus according to an aspect of the present disclosure includes a pixel array in which a first pixel in which an OCL (On Chip Lens) of a standard size is formed and a second pixel in which an OCL of a size different from the standard size is formed are present mixedly, and a correction section that corrects a pixel value of the first pixel that is positioned in the vicinity of the second pixel among the first pixels on the pixel array. The present disclosure can be applied to, for example, a CMOS image sensor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is continuation of U.S. Application Serial No.17/090,666, filed Nov. 5, 2020, which is a continuation of U.S.Application Serial No. 16/312,417 filed Dec. 21, 2018, now U.S. Pat. No.10,863,124, which is a national stage application under 35 U.S.C. 371and claims the benefit of PCT Application No. PCT/JP2017/022986 havingan international filing date of Jun. 22, 2017, which designated theUnited States, which PCT application claimed the benefit of JapanesePatent Application No. 2016-133966 filed Jul. 06, 2016, the entiredisclosures of each of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION Technical Field

The present disclosure relates to a solid-state image pickup apparatus,a correction method, and an electronic apparatus, and in particular,relates to a solid-state image pickup apparatus, a correction method,and an electronic apparatus, enabled to correct a pixel value of astandard pixel for generating a color signal that is present on theperiphery of a phase difference detection pixel for generating a phasedifference signal.

Background Art

As a method of an AF (Auto Focus) system of a digital camera etc., animaging plane phase difference AF is known.

In an image sensor for realizing the imaging plane phase difference AF,a partially shaded phase difference detection pixel that outputs a pixelvalue for generating a phase difference signal is arranged on a pixelarray in which a standard pixel that outputs a color signal is arrangedvertically and horizontally. Note, however, that in a case of using thepartially shaded phase difference detection pixel, since an appropriatephase difference signal is not obtained at the time of low-lightintensity, an AF performance may be decreased.

As an image sensor capable of suppressing such a situation, aconfiguration in which the phase difference detection pixel having a 2×1OCL (On Chip Lens) structure (hereinafter, simply referred to as a 2×1OCL pixel) is discretely arranged between standard pixels has beenproposed. The OCL is a lens that is formed on the side of receivinglight of pixels in order to improve light collection efficiency of thepixels. Further, the phase difference detection pixel having the 2×1 OCLstructure includes an OCL having a horizontally long shape (orvertically long shape) for two pixels of the standard pixel on areceiving light surface. Further, light passing through the right sideand the left side of the OCL, respectively, is received by differentreceiving light elements within the phase difference detection pixel tothereby generate a phase difference signal from a pixel value to beobtained.

Note that, in a case where an output from the image sensor in which a2×1 OCL pixel is mounted mixedly on the pixel array is used as imagedata, appropriate interpolation processing needs to be performed toobtain a color signal corresponding to a position of the 2×1 OCL pixel.As the interpolation processing, for example, direction interpolationprocessing in which directivity in a peripheral area of the 2×1 OCLpixel is detected and interpolated, correlation interpolation processingin which a level of each color in the peripheral area is used, or thelike can be used.

Citation List Patent Literature

PTL 1: Japanese Patent Laid-Open No. 1989-216306

SUMMARY OF THE INVENTION Technical Problem

However, even if a correction is performed on a position of the 2×1 OCLpixel, in a case where the position is a flat portion of the image, anapparent uncomfortable feeling may be allowed to be generated near tothe 2×1 OCL pixel due to a color signal that is an output of thestandard pixel adjacent to the 2×1 OCL pixel.

Specifically, in a manufacturing process of the image sensor in whichthe 2×1 OCL pixel is mounted mixedly, the 2×1 OCL pixel is different, ina shape of the formed OCL, from the standard pixel on the peripherythereof. Therefore, even an OCL shape of the standard pixel on theperiphery of the 2×1 OCL pixel is slightly deformed from an originalshape. Further, a size of the standard pixel is increased or decreasedand a symmetry of the shape is broken.

Such a deformation of the OCL of the standard pixel exerts an influenceon a sensitivity of the standard pixel and the obtained color signal isincreased or decreased minutely from a value that ought to be originallyobtained. As a result, uncomfortable feeling is caused in the flatportion.

Such a deformation of the OCL of the standard pixel exerts an influenceon a sensitivity of the standard pixel and the obtained color signal isincreased or decreased minutely from a value that ought to be originallyobtained. As a result, uncomfortable feeling is caused in the flatportion.

The present disclosure is performed by considering such a situation andaims at suppressing an apparent uncomfortable feeling of an image outputfrom a solid-state image pickup apparatus in which pixels havingdifferent OCL shapes are mounted mixedly.

Solution to Problem

According to a first aspect of the present disclosure, a solid-stateimage pickup apparatus includes: a pixel array in which a first pixel inwhich an OCL (On Chip Lens) of a standard size is formed and a secondpixel in which an OCL of a size different from the standard size isformed are present mixedly; and a correction section that corrects apixel value of the first pixel that is positioned in the vicinity of thesecond pixel among the first pixels on the pixel array.

The correction section may correct a pixel value of a third pixel thatis positioned in the vicinity of the second pixel and in which a shapeof the OCL is deformed from an original standard size among the firstpixels on the pixel array.

An OCL having a size larger than the standard size may be formed in thesecond pixel.

The correction section may calculate a correction value that is replacedwith a pixel value of the third pixel as a correction target by using apixel value of the first pixel that is positioned in the vicinity of thethird pixel as the correction target.

The correction section may decide a threshold for determining whether ornot the calculated correction value is applied.

The correction section may decide the threshold by using a parameterthat is different depending on a positional relationship between thethird pixel as the correction target and the second pixel.

The correction section may decide the threshold by using a parameterthat is different depending on a position in an image of the third pixelas the correction target.

The correction section may determine whether or not the calculatedcorrection value is applied on the basis of a flatness in a peripheralarea of the third pixel as the correction target.

The second pixel may be a phase difference detection pixel.

According to the first aspect of the present disclosure, the solid-stateimage pickup apparatus may further include an interpolation section thatinterpolates a pixel value corresponding to a position of the secondpixel.

According to a first aspect of the present disclosure, a method forcorrecting a solid-state image pickup apparatus including a pixel arrayin which a first pixel in which an OCL (On Chip Lens) of a standard sizeis formed and a second pixel in which an OCL of a size different fromthe standard size is formed are present mixedly and a correction sectionthat corrects an output of the pixel array, the method includes acorrection step of correcting, by the correction section, a pixel valueof the first pixel that is positioned in the vicinity of the secondpixel among the first pixels on the pixel array.

According to the first aspect of the present disclosure, a pixel valueof the first pixel that is positioned in the vicinity of the secondpixel among the first pixels on the pixel array is corrected.

According to a second aspect of the present disclosure, an electronicapparatus includes a solid-state image pickup apparatus to be mountedthereon, in which the solid-state image pickup apparatus includes: apixel array in which a first pixel in which an OCL (On Chip Lens) of astandard size is formed and a second pixel in which an OCL of a sizedifferent from the standard size is formed are present mixedly; and acorrection section that corrects a pixel value of the first pixel thatis positioned in the vicinity of the second pixel among the first pixelson the pixel array.

According to the second aspect of the present disclosure, the pixelvalue of the first pixel that is positioned in the vicinity of thesecond pixel among the first pixels on the pixel array is corrected bythe solid-state image pickup apparatus.

Advantageous Effects of Invention

According to the first and second aspects of the present disclosure, itis possible to correct a pixel value of a second pixel in which a shapeof an OCL is deformed from a standard shape, which is positioned on theperiphery of a first pixel and in which the shape of the OCL isdifferent from the standard shape.

Further, according to the first and second aspects of the presentdisclosure, it is possible to suppress an apparent uncomfortable feelingthat may occur on an image.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram depicting an arrangement example of a color filter.

FIG. 2 is a diagram depicting shapes of OCLs of a standard pixel, adeformed pixel, and an irregular pixel.

FIG. 3 is a cross-sectional diagram depicting a specific example of adeformed OCL of the deformed pixel.

FIG. 4 is a top diagram depicting a specific example of the deformed OCLof the deformed pixel.

FIG. 5 is a block diagram depicting a configuration example of an imagesensor to which the present disclosure is applied.

FIG. 6 is a block diagram depicting a configuration example of adeformed pixel correction section.

FIG. 7 is a diagram depicting an allowable range in a case where thedeformed pixel is corrected.

FIG. 8 is a diagram depicting an allowable range in the case where thedeformed pixel is corrected.

FIG. 9 is a diagram depicting a positional relationship between theirregular pixel and the deformed pixel.

FIG. 10 is a diagram depicting an area in which a parameter isindividually defined.

FIG. 11 is a flowchart depicting deformed pixel correction processing.

FIG. 12 is a block diagram depicting an example of a schematicconfiguration of an in-vivo information acquisition system.

FIG. 13 is a block diagram depicting an example of schematicconfiguration of a vehicle control system.

FIG. 14 is a diagram of assistance in explaining an example ofinstallation positions of an outside-vehicle information detectingsection and an imaging section.

DETAILED DESCRIPTION OF THE INVENTION Description of Embodiments

Hereinafter, a best mode (hereinafter, referred to as an embodiment) forcarrying out the present disclosure will be described in detail withreference to the drawings.

Note that, in the following description, a pixel having an OCL occupyingan area for one pixel for generating a color signal is referred to as astandard pixel among a plurality of pixels arranged in a pixel array ofan image sensor (solid-state image pickup apparatus). Further, a phasedifference detection pixel having an OCL (i.e., 2×1 OCL) occupying anarea for two pixels that output a pixel value for generating a phasedifference signal is also referred to as a 2×1 OCL pixel or an irregularpixel. In addition, among the standard pixels, a pixel in which a shapeof the OCL is affected by a formation of the 2×1 OCL and is deformedfrom an original shape is referred to as a deformed pixel.

The image sensor according to the present embodiment interpolates acolor signal corresponding to a position of the 2×1 OCL pixel and, atthe same time, corrects a color signal of the deformed pixel. Theprocess permits an apparent uncomfortable feeling on the periphery ofthe 2×1 OCL pixel in a flat portion to be suppressed.

Note that shapes of the phase difference detection pixel and an OCLthereof are not limited to 2×1. When the shape is other than 1×1 that isa shape of the standard pixel and an OCL thereof, it may be, forexample, 1×2, 2×2, or the like. In addition, a pixel having two or morekinds of irregular OCLs may be mounted mixedly on the pixel array.

Configuration Example of Image Sensor According to the PresentEmbodiment

FIG. 1 depicts an array example of a color filter in the pixel array 11that is mounted on the image sensor 10 (depicted in FIG. 5 ) accordingto the present embodiment.

In this arrangement example, the standard pixels and the deformed pixelsbasically form a Bayer array. Note that, R (RED), G (GREEN), and B(BLUE) in the figure indicate an arrangement of the standard pixels(including the deformed pixels) that are covered by color filters of R,G, and B, respectively.

Further, PD in the figure indicates an arrangement of the 2×1 OCL pixeland is discretely arranged on the entire pixel array 11. Note that, itis assumed that the PD occupies an area for two pixels of G and B in theBayer array and is covered with a color filter of G. Note that, a colorof the color filter covering the 2×1 OCL pixel is not limited to G;further, it may be R or B.

Note that, the arrangement example of the standard pixel is not limitedto the above-described Bayer array; further, it may be applied to otherarrays.

Next, the shape of the OCL of the standard pixels, deformed pixels, andirregular pixels (2×1 OCL pixels) that are mounted mixedly on the pixelarray 11 will be described with reference to FIGS. 2 to 4 .

FIG. 2 depicts a cross-sectional shape in a transverse direction of theOCLs of the standard pixel, the deformed pixel, and the irregular pixel.FIG. 3 is a cross-sectional diagram depicting a specific example of theOCL of the deformed pixel. FIG. 4 is a top diagram depicting a specificexample of the OCL of the deformed pixel.

As depicted in FIG. 2 , the OCL of the standard pixel occupies an areafor one pixel, which is circular. By contrast, the OCL of the irregularpixel occupies an area for two pixels, which is horizontally long andhas an elliptical shape. In the irregular pixel, incident light thatpasses through a left-side portion and a right-side portion of the OCLhaving an elliptical shape, respectively, is incident on differentphotoelectric conversion sections (not depicted) to thereby generate aphase difference signal.

As apparent from the cross-sectional shape depicted in FIG. 2 , it isunderstood that a shape of the OCL is slightly deformed from an originalshape in the deformed pixel adjacent to the irregular pixel. In amanufacturing process of the pixel array 11, the above fact is caused byan influence at the time of forming the 2×1 OCL of the irregular pixel.

Note that, as depicted in FIG. 2 , in a case where the irregular pixelis lower in a height of the OCL than the standard pixel, a lightcollection efficiency tends to be reduced to slightly decreasesensitivity.

Further, as depicted in FIG. 3 , a symmetry may be broken in the OCL ofthe deformed pixel. In this case, the sensitivity fluctuates due to anangle of light that is incident on the OCL. Therefore, the sensitivityis increased or decreased depending on a position of the deformed pixelin the image.

Further, as depicted in FIG. 4 , the deformed pixel in which the shapeof the OCL has changed may appear not only in a transverse direction ofthe irregular pixel but also in a longitudinal direction thereof.Therefore, the above deformed pixel is also a correction target of thepixel value.

In examples depicted in FIGS. 2 and 4 , pixels adjacent to the left,right, top and bottom of the irregular pixel are deformed pixels.Further, an influence at the time of forming the 2×1 OCL of theirregular pixel in the manufacturing process may extend to pixels in awider range. In such a case, all pixels between the irregular pixel andthe standard pixels in which the OCL is not deformed just have to be setas a deformed pixel to be a processing target of deformed pixelcorrection processing described below.

Next, FIG. 5 depicts a configuration example of an image sensor(solid-state image pickup apparatus) according to an embodiment of thepresent disclosure. The image sensor 10 is mounted on an electronicapparatus having an imaging plane phase difference AF function. Notethat the image sensor 10 may be a surface irradiation type or a backsurface irradiation type. In addition, the image sensor 10 may be alaminated type including a plurality of substrates.

The image sensor 10 includes a pixel array 11, a phase differencedetection section 12, an irregular pixel interpolation section 13, an AFcontrol section 14, a deformed pixel correction section 15, a defectivepixel correction section 16, and a camera signal processing section 17.

The pixel array 11 outputs each output of the standard pixel, deformedpixel, and irregular pixel mounted mixedly as depicted in FIGS. 1 and 2to the phase difference detection section 12. Note that, in the outputof the pixel array 11 in this stage, the standard pixel and the deformedpixel have any one of the color signals of R, G, and B and the irregularpixel has a pixel value for generating a phase difference signal.

From among the outputs of the pixel array 11, on the basis of a pixelvalue of the irregular pixel, the phase difference detection section 12detects a phase difference signal corresponding to a deviation in afocus and outputs the phase difference signal to the irregular pixelinterpolation section 13 and the AF control section 14. Further, thephase difference detection section 12 outputs a color signal of thestandard pixel and the deformed pixel to the irregular pixelinterpolation section 13.

On the basis of an input from a preceding stage, the irregular pixelinterpolation section 13 interpolates a color signal in a position ofthe phase difference pixel (irregular pixel) by using a predeterminedinterpolation method (a direction interpolation method in whichdirectivity in a peripheral area is detected and interpolated, acorrelation interpolation method in which a level of each color in aperipheral area is used, or the like). Further, the irregular pixelinterpolation section 13 outputs a color signal interpolated on aposition of the irregular pixel, and a color signal of the standardpixel and the deformed pixel to the deformed pixel correction section15.

On the basis of the detected phase difference signal, the AF controlsection 14 generates a lens control signal for driving a focus lens andoutputs the lens control signal to a lens drive section (either is notdepicted).

On the basis of an input from the preceding stage, the deformed pixelcorrection section 15 performs correction processing on a color signalof the deformed pixel. Details of the correction processing will bedescribed below. Further, the deformed pixel correction section 15outputs a color signal interpolated on a position of the irregularpixel, a corrected color signal of the deformed pixel, and a colorsignal of the standard pixel to the defective pixel correction section16.

On the basis of an input from the preceding stage, the defective pixelcorrection section 16 interpolates a color signal of the defectivepixel. Further, the defective pixel correction section 16 outputs acolor signal interpolated on a position of the irregular pixel, acorrected color signal of the deformed pixel, a color signal of thestandard pixel, and an interpolated color signal of the defective pixelto the camera signal processing section 17.

The camera signal processing section 17 performs predetermined camerasignal processing (white balance processing, demosaic processing, linearmatrix processing, gamma correction processing, or the like) on any ofthe color signals of R, G, and B included in each pixel. Further, thecamera signal processing section 17 outputs an RGB image in which eachpixel obtained as a result has all color signals of R, G, and B to thefollowing stage.

Operation of the Image Sensor 10

In the image sensor 10, the phase difference detection signal isdetected by using the phase difference detection section 12 from theoutput of the pixel array 11 to thereby perform AF control. Further, acolor signal is interpolated on a position of the irregular pixel byusing the irregular pixel interpolation section 13, a color signal ofthe deformed pixel is corrected by using the deformed pixel correctionsection 15, and a color signal of the defective pixel is interpolated byusing the defective pixel correction section 16. Further, on the basisof the correction results, the RGB image is generated and output byusing the camera signal processing section 17.

Note that, a circuit configuration and processing sequence of theirregular pixel interpolation section 13, the deformed pixel correctionsection 15, and the defective pixel correction section 16 need not beconnected in series as depicted in the figure. Further, the abovesections may be connected in parallel to each other and may be allowedto perform each processing at the same time. Through the process, it ispossible to more quickly output the RGB image from the image sensor 10.

Detailed Configuration Example of the Deformed Pixel Correction Section15

Next, FIG. 6 depicts a detailed configuration example of the deformedpixel correction section 15. The deformed pixel correction section 15includes a correction value calculation section 21, a flatnessdetermination section 22, a threshold calculation section 23, an averagevalue calculation section 24, and a correction section 25.

The correction value calculation section 21 calculates a correctionvalue for each deformed pixel in which a pixel value is corrected, andoutputs the correction value to the correction section 25. Specifically,among pixels that are present in a predetermined area centering on thedeformed pixel, the correction value calculation section 21 calculatesas a correction value an average value of pixel values of the deformedpixel, excluding the irregular pixels and the other deformed pixels, andthe standard pixel. Alternatively, the correction value calculationsection 21 detects a directivity of a texture on the periphery of thedeformed pixel and generates the correction value by using a directioninterpolation using pixels along the detected direction. Note that, amethod for calculating the correction value in the correction valuecalculation value 21 is arbitrary and any method except theabove-described calculation method may be used.

The flatness determination section 22 determines flatness on theperiphery of the deformed pixel to correct the pixel value and outputs adetermination result to the correction section 25. Specifically, theflatness determination section 22 calculates a frequency distributionregarding differences of the pixel values of the pixels that are presentin a predetermined area centering on the deformed pixel. Then, if a rateof the pixels is high in which the calculated difference is equal to orsmaller than the threshold, it is determined that the deformed pixel ispresent in a flat area or in an area having a simple texture such as astaircase pattern. Alternatively, a two-dimensional high-pass filter maybe applied thereto and flatness may be determined on the basis ofwhether or not a value of the output is equal to or smaller than thethreshold. Note that a method for determining the flatness by theflatness determination section 22 is arbitrary and any method except theabove-described determination method may be used.

The threshold calculation section 23 calculates a lower limit thresholdand upper limit threshold for determining whether a correction isperformed by using the calculated correction value, and outputs thelower limit threshold and the upper limit threshold to the correctionsection 25. Specifically, in a case where the calculated correctionvalue is present between the lower limit threshold and the upper limitthreshold, the correction is performed by using the calculatedcorrection value. By contrast, in a case where the calculated correctionvalue is not present between the lower limit threshold and the upperlimit threshold, the correction is not performed.

The average value calculation section 24 calculates an average value ofthe pixel values of the pixels that are present in a predetermined areacentering on the deformed pixel to correct the pixel value and outputsthe average value to the threshold calculation section 23. Specifically,the average value calculation section 24 calculates an average value ofa luminance signal converted by using all of the color signals of R, G,and B, or calculates an addition average value by using only a colorsignal of G.

The correction section 25 corrects a pixel value (color signal) of thedeformed pixel on the basis of an input from the correction valuecalculation section 21, the flatness determination section 22, and thethreshold calculation section 23.

Here, a calculation of the upper limit threshold and lower limitthreshold by the threshold calculation section 23 will be described indetail.

FIGS. 7 and 8 depict an allowable range in a case where the pixel valueof the deformed pixel is replaced with the correction value. Note that,in FIG. 7 , a horizontal axis is set as the pixel value of the deformedpixel and a vertical axis is set as the correction value thereof. InFIG. 8 , the horizontal axis is set as the pixel value of the deformedpixel and the vertical axis is set as a change allowable range at thetime of correcting the pixel value.

Fluctuation of the pixel value of the deformed pixel is caused by achange in a sensitivity due to a deformation of the OCL. Therefore, theabove fluctuation is not related to the amount of light incident on theadjacent irregular pixel, namely, the pixel value of the irregularpixel. The amount of fluctuation of the pixel value in the deformedpixel increases or decreases in a manner proportional to the pixelvalue.

Therefore, in a case where correcting the pixel value that fluctuates asdescribed above, it is desirable to use an average value of the pixelvalues of the pixels in a predetermined area including the deformedpixel to stabilize processing. When the average value is small, afluctuation width of the pixel value after the correction is made small.By contrast, when the average value is large, the fluctuation width ofthe pixel value after the correction is made large.

The fluctuation width from the pixel value before the correction of thedeformed pixel is calculated by using the fluctuation width ofst at thetime when the pixel value before the correction of the deformed pixel isequal to 0 and a gradient grad of a component proportional to the pixelvalue before the correction of the deformed pixel. As a result, theupper limit threshold TH_UP and lower limit threshold TH_LW for thecorrection value are defined by the following formula (1).

$\begin{array}{l}{\text{TH\_UP} = \text{Pix} + \text{AVE} \times \text{grad\_up} + \text{ofst}} \\{\text{TH\_LW} = \text{Pix} + \text{AVE} \times \text{grad\_lw} + \text{ofst}}\end{array}$

Note, however, that Pix in formula (1) is a pixel value before thecorrection of the deformed pixel. AVE is an average value of the pixelvalues of the pixels in a predetermined area including the deformedpixel. Further, grad_up is a gradient on the upper side. Further,grad_lw is a gradient on the lower side.

Note that, a deformation degree of the OCL of the deformed pixel is notnecessarily the same and may vary depending on a positional relationship(any position to the left, right, top and bottom of the irregular pixel)between the deformed pixel and the irregular pixel, the manufacturingprocess, or a pixel size. Therefore, regarding the gradients grad_up andgrad_lw and the fluctuation width ofst that are parameters forcalculating the above-described upper limit threshold TH_UP and lowerlimit threshold TH_LW, as depicted in FIG. 9 , six kinds of values aredefined in accordance with a position of the deformed pixelcorresponding to the irregular pixel

In particular, in a case where asymmetric deformation is seen in the OCLof the deformed pixel, the sensitivity may increase or decreasedepending on a position in the image of the deformed pixel. In a casewhere a lateral symmetry is broken, the entire image just has to bedivided into three in the longitudinal direction, for example, asdepicted in FIG. 10 . Further, in each of the three-divided areas, theabove-described parameter just has to be defined so as to be adjusted.Further, in a case where an asymmetry is seen to the left, right, topand bottom of the deformed pixel, the entire image just has to bedivided in the longitudinal direction and in the transverse directionand the above-described parameter just has to be defined in each of thedivided areas.

Deformed Pixel Correction Processing by the Deformed Pixel CorrectionSection 15

Next, FIG. 11 is a flowchart depicting the deformed pixel correctionprocessing by the deformed pixel correction section 15.

The deformed pixel correction processing is performed by sequentiallyassigning the deformed pixel to a processing target. As a prerequisite,a color signal interpolated on a position of the irregular pixel fromthe irregular pixel interpolation section 13 of the preceding stage anda color signal of the standard pixel and the deformed pixel are assumedto be supplied to the deformed pixel correction section 15. Further, theoutput of the correction value calculation section 21, the flatnessdetermination section 22, and the threshold calculation section 23 isassumed to be input to the correction section 25 of the deformed pixelcorrection section 15.

At step S1, on the basis of the input from the flatness determinationsection 22, the correction section 25 determines whether or not thedeformed pixel as the processing target is positioned in a flat area orin an area having a simple texture. If it is determined that thedeformed pixel as the processing target is positioned in the flat areaor in the area having the simple texture, the process proceeds to stepS2.

At step S2, on the basis of the input from the correction valuecalculation section 21 and the threshold calculation section 23, thecorrection section 25 determines whether or not the calculatedcorrection value is positioned between the lower limit threshold TH_LWand the upper limit threshold TH_UP. If it is determined that thecalculated correction value is positioned between the lower limitthreshold TH_LW and the upper limit threshold TH_UP, the processproceeds to step S3.

At step S3, the correction section 25 replaces the pixel value of thedeformed pixel as the processing target with a correction value inputfrom the correction value calculation section 21. Then, the deformedpixel correction processing on the processing target ends.

Note that, at step S1, if it is determined that the deformed pixel asthe processing target is neither positioned in the flat area nor in thearea having the simple texture, a sensitivity influence of the deformedpixel is inconspicuous and therefore the correction is not performed.Therefore, step S3 is skipped and the deformed pixel correctionprocessing ends.

Further, at step S2, even if it is determined that the calculatedcorrection value is not positioned between the lower limit thresholdTH_LW and the upper limit threshold TH_UP, the correction in which thepixel value of the deformed pixel is replaced with the calculatedcorrection value may be an erroneous correction, and therefore thecorrection is not performed. Therefore, step S3 is skipped and thedeformed pixel correction processing ends.

As described above, the image sensor 10 according to the presentembodiment not only interpolates a color signal in a position of theirregular pixel (2×1 OCL pixel) but also corrects the pixel value of thedeformed pixel in which the OCL on the periphery of the irregular pixelis deformed from an original shape. Therefore, it is possible tosuppress an apparent uncomfortable feeling on the phase differencedetection pixel and the periphery thereof in the image.

Further, it is possible to adjust an upper limit and lower limit of thecorrection value in accordance with a relative positional relationshipof the deformed pixel to the irregular pixel and a position of thedeformed pixel in the image. Therefore, it is possible to suppress anovercorrection and more appropriately correct the pixel value of thedeformed pixel.

Note that, in the present embodiment, a purpose of the irregular pixelhaving the 2×1 OCL is for a phase difference detection for the phasedifference AF control. Further, a purpose of the irregular pixel that isdifferent, in a shape of the OCL, from the standard pixel is not limitedto the phase difference detection.

First Application Example

The technology according to the present disclosure can be applied tovarious products. For example, the technology according to the presentdisclosure may be applied to an in-vivo information acquisition systemof a patient using a capsule type endoscope.

FIG. 12 is a view depicting an example of a schematic configuration ofan in-vivo information acquisition system 5400 to which the technologyaccording to an embodiment of the present disclosure can be applied.Referring to FIG. 12 , the in-vivo information acquisition system 5400includes a capsule type endoscope 5401, and an external controllingapparatus 5423 which integrally controls operation of the in-vivoinformation acquisition system 5400. Upon inspection, the capsule typeendoscope 5401 is swallowed by a patient. The capsule type endoscope5401 has an image pickup function and a wireless communication function.For a period of time before the capsule type endoscope 5401 isdischarged naturally from the patient, while it moves in the inside ofan organ such as the stomach or the intestines by peristaltic motion, itsuccessively picks up an image in the inside of each organ (hereinafterreferred to as in-vivo image) at predetermined intervals andsuccessively transmits information of the in-vivo images in wirelessfashion to the external controlling apparatus 5423 located outside thebody. The external controlling apparatus 5423 generates image data fordisplaying the in-vivo images on a display apparatus (not depicted) onthe basis of the information of the received in-vivo images. In thismanner, in the in-vivo information acquisition system 5400, a picked upimage illustrating a state of the inside of the body of the patient canbe obtained at any time after the capsule type endoscope 5401 isswallowed until it is discharged.

A configuration and functions of the capsule type endoscope 5401 and theexternal controlling apparatus 5423 are described in more detail. Asdepicted, the capsule type endoscope 5401 has functions of a lightsource unit 5405, an image pickup unit 5407, an image processing unit5409, a wireless communication unit 5411, a power feeding unit 5415, apower supply unit 5417, a state detection unit 5419 and a control unit5421 incorporated in a housing 5403 of the capsule type.

The light source unit 5405 includes a light source such as, for example,a light emitting diode (LED) and irradiates light upon an image pickupfield of view of the image pickup unit 5407.

The image pickup unit 5407 includes an image pickup element and anoptical system formed from a plurality of lenses provided at a precedingstage to the image pickup element. Reflected light (hereinafter referredto as observation light) of light irradiated upon a body tissue which isan observation target is condensed by the optical system and enters theimage pickup element. The image pickup element receives andphotoelectrically converts the observation light to generate an electricsignal corresponding to the observation light, namely, an image signalcorresponding to an observation image. The image signal generated by theimage pickup unit 5407 is provided to the image processing unit 5409. Itis to be noted that, as the image pickup element of the image pickupunit 5407, various known image pickup elements such as a complementarymetal oxide semiconductor (CMOS) image sensor or a charge coupled device(CCD) image sensor may be used.

The image processing unit 5409 includes a processor such as a centralprocessing unit (CPU) or a graphics processing unit (GPU) and performsvarious signal processes for an image signal generated by the imagepickup unit 5407. The signal processes may be minimal processes fortransmitting an image signal to the external controlling apparatus 5423(for example, compression of image data, conversion of the frame rate,conversion of the data rate, and/or conversion of the format). Since theimage processing unit 5409 is configured so as to perform only theminimal processes, the image processing unit 5409 can be implemented ina smaller size with lower power consumption. Therefore, the imageprocessing unit 5409 is suitable for the capsule type endoscope 5401.However, if the space in the housing 5403 or the power consumptionaffords, then the image processing unit 5409 may perform a furthersignal process (for example, a noise removal process or some other imagequality improving process). The image processing unit 5409 provides animage signal, for which the signal processes have been performed, as RAWdata to the wireless communication unit 5411. It is to be noted that,when information regarding a state (motion, posture or the like) of thecapsule type endoscope 5401 is acquired by the state detection unit5419, the image processing unit 5409 may provide an image signal in atied manner with the information to the wireless communication unit5411. This makes it possible to associate the position inside the bodyat which an image is picked up, an image pickup direction of the imageor the like with the picked up image.

The wireless communication unit 5411 includes a communication apparatuswhich can transmit and receive various kinds of information to and fromthe external controlling apparatus 5423. The communication apparatusincludes an antenna 5413, a processing circuit which performs amodulation process and so forth for transmission and reception of asignal, and so forth. The wireless communication unit 5411 performs apredetermined process such as a modulation process for an image signalfor which the signal processes have been performed by the imageprocessing unit 5409, and transmits the resulting image signal to theexternal controlling apparatus 5423 through the antenna 5413. Further,the wireless communication unit 5411 receives a control signal relatingto driving control of the capsule type endoscope 5401 from the externalcontrolling apparatus 5423 through the antenna 5413. The wirelesscommunication unit 5411 provides the received control signal to thecontrol unit 5421.

The power feeding unit 5415 includes an antenna coil for powerreception, a power regeneration circuit for regenerating electric powerfrom electric current generated in the antenna coil, a voltage boostercircuit and so forth. The power feeding unit 5415 generates electricpower using the principle of non-contact charging. Specifically, if amagnetic field (electromagnetic wave) of a predetermined frequency isprovided from the outside to the antenna coil of the power feeding unit5415, then induced electromotive force is generated in the antenna coil.The electromagnetic wave may be a carrier transmitted from the externalcontrolling apparatus 5423 through an antenna 5425. Electric power isregenerated from the induced electromotive force by the powerregeneration circuit, and the potential of the electric power issuitably adjusted by the voltage booster circuit to generate electricpower for charging. The electric power generated by the power feedingunit 5415 is stored into the power supply unit 5417.

The power supply unit 5417 includes a secondary battery and storeselectric power generated by the power feeding unit 5415. In FIG. 12 , inorder to avoid complicated illustration, an arrow mark indicative of asupplying destination of electric power from the power supply unit 5417and so forth are not depicted. However, electric power stored in thepower supply unit 5417 is supplied to the light source unit 5405, theimage pickup unit 5407, the image processing unit 5409, the wirelesscommunication unit 5411, the state detection unit 5419 and the controlunit 5421 and can be used for driving of them.

The state detection unit 5419 includes a sensor for detecting a state ofthe capsule type endoscope 5401 such as an acceleration sensor and/or agyro sensor. The state detection unit 5419 can acquire informationrelating to a state of the capsule type endoscope 5401 from a result ofdetection by the sensor. The state detection unit 5419 provides theacquired information regarding a state of the capsule type endoscope5401 to the image processing unit 5409. The image processing unit 5409can tie the information regarding a state of the capsule type endoscope5401 with an image signal as described hereinabove.

The control unit 5421 includes a processor such as a CPU and operates inaccordance with a predetermined program to integrally control operationof the capsule type endoscope 5401. The control unit 5421 suitablycontrols driving of the light source unit 5405, the image pickup unit5407, the image processing unit 5409, the wireless communication unit5411, the power feeding unit 5415, the power supply unit 5417 and thestate detection unit 5419 in accordance with a control signaltransmitted thereto from the external controlling apparatus 5423 toimplement such functions of the components as described above.

The external controlling apparatus 5423 may be a processor such as a CPUor a GPU, a microcomputer or a control board in which a processor and astorage element such as a memory are mixedly incorporated. The externalcontrolling apparatus 5423 is configured such that it has an antenna5425 and can transmit and receive various kinds of information to andfrom the capsule type endoscope 5401 through the antenna 5425.Specifically, the external controlling apparatus 5423 transmits acontrol signal to the control unit 5421 of the capsule type endoscope5401 to control operation of the capsule type endoscope 5401. Forexample, an irradiation condition of light upon an observation target ofthe light source unit 5405 can be changed in accordance with a controlsignal from the external controlling apparatus 5423. Further, an imagepickup condition (for example, a frame rate, an exposure value or thelike of the image pickup unit 5407) can be changed in accordance with acontrol signal from the external controlling apparatus 5423. Further,the substance of processing by the image processing unit 5409 or acondition for transmitting an image signal from the wirelesscommunication unit 5411 (for example, a transmission interval, atransmission image number or the like) may be changed in accordance witha control signal from the external controlling apparatus 5423.

Further, the external controlling apparatus 5423 performs various imageprocesses for an image signal transmitted from the capsule typeendoscope 5401 to generate image data for displaying a picked up in-vivoimage on the display apparatus. As the image processes, various knownsignal processes may be performed such as, for example, a developmentprocess (demosaic process), an image quality improving process(bandwidth enhancement process, super-resolution process, noisereduction (NR) process, and/or image stabilization process) and/or anenlargement process (electronic zoom process) or the like. The externalcontrolling apparatus 5423 controls driving of the display apparatus(not depicted) to cause the display apparatus to display a picked upin-vivo image on the basis of generated image data. Alternatively, theexternal controlling apparatus 5423 may control a recording apparatus(not depicted) to record generated image data or control a printingapparatus (not depicted) to output generated image data by printing.

As described above, an example of the in-vivo information acquisitionsystem 5400 to which the technology according to the present disclosureis applicable is described. The technology according to the presentdisclosure can be preferably applied to the image pickup unit 5407 inthe configuration described above.

Second Application Example

For example, the technology according to the present disclosure may beimplemented as apparatuses mounted on any type of mobile objects such asautomobiles, electric vehicles, hybrid electric vehicles, motorcycles,bicycles, personal mobilities, airplanes, drones, ships, robots,construction machines, and agricultural machines (tractors).

FIG. 13 is a block diagram depicting an example of schematicconfiguration of a vehicle control system 7000 as an example of a mobilebody control system to which the technology according to an embodimentof the present disclosure can be applied. The vehicle control system7000 includes a plurality of electronic control units connected to eachother via a communication network 7010. In the example depicted in FIG.13 , the vehicle control system 7000 includes a driving system controlunit 7100, a body system control unit 7200, a battery control unit 7300,an outside-vehicle information detecting unit 7400, an in-vehicleinformation detecting unit 7500, and an integrated control unit 7600.The communication network 7010 connecting the plurality of control unitsto each other may, for example, be a vehicle-mounted communicationnetwork compliant with an arbitrary standard such as controller areanetwork (CAN), local interconnect network (LIN), local area network(LAN), FlexRay, or the like.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various kinds of programs; a storagesection that stores the programs executed by the microcomputer,parameters used for various kinds of operations, or the like; and adriving circuit that drives various kinds of control target devices.Each of the control units further includes: a network interface (I/F)for performing communication with other control units via thecommunication network 7010; and a communication I/F for performingcommunication with a device, a sensor, or the like within and withoutthe vehicle by wire communication or radio communication. A functionalconfiguration of the integrated control unit 7600 illustrated in FIG. 13includes a microcomputer 7610, a general-purpose communication I/F 7620,a dedicated communication I/F 7630, a positioning section 7640, a beaconreceiving section 7650, an in-vehicle device I/F 7660, a sound/imageoutput section 7670, a vehicle-mounted network I/F 7680, and a storagesection 7690. The other control units similarly include a microcomputer,a communication I/F, a storage section, and the like.

The driving system control unit 7100 controls the operation of devicesrelated to the driving system of the vehicle in accordance with variouskinds of programs. For example, the driving system control unit 7100functions as a control device for a driving force generating device forgenerating the driving force of the vehicle, such as an internalcombustion engine, a driving motor, or the like, a driving forcetransmitting mechanism for transmitting the driving force to wheels, asteering mechanism for adjusting the steering angle of the vehicle, abraking device for generating the braking force of the vehicle, and thelike. The driving system control unit 7100 may have a function as acontrol device of an antilock brake system (ABS), electronic stabilitycontrol (ESC), or the like.

The driving system control unit 7100 is connected with a vehicle statedetecting section 7110. The vehicle state detecting section 7110, forexample, includes at least one of a gyro sensor that detects the angularvelocity of axial rotational movement of a vehicle body, an accelerationsensor that detects the acceleration of the vehicle, and sensors fordetecting an amount of operation of an accelerator pedal, an amount ofoperation of a brake pedal, the steering angle of a steering wheel, anengine speed or the rotational speed of wheels, and the like. Thedriving system control unit 7100 performs arithmetic processing using asignal input from the vehicle state detecting section 7110, and controlsthe internal combustion engine, the driving motor, an electric powersteering device, the brake device, and the like.

The body system control unit 7200 controls the operation of variouskinds of devices provided to the vehicle body in accordance with variouskinds of programs. For example, the body system control unit 7200functions as a control device for a keyless entry system, a smart keysystem, a power window device, or various kinds of lamps such as aheadlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or thelike. In this case, radio waves transmitted from a mobile device as analternative to a key or signals of various kinds of switches can beinput to the body system control unit 7200. The body system control unit7200 receives these input radio waves or signals, and controls a doorlock device, the power window device, the lamps, or the like of thevehicle.

The battery control unit 7300 controls a secondary battery 7310, whichis a power supply source for the driving motor, in accordance withvarious kinds of programs. For example, the battery control unit 7300 issupplied with information about a battery temperature, a battery outputvoltage, an amount of charge remaining in the battery, or the like froma battery device including the secondary battery 7310. The batterycontrol unit 7300 performs arithmetic processing using these signals,and performs control for regulating the temperature of the secondarybattery 7310 or controls a cooling device provided to the battery deviceor the like.

The outside-vehicle information detecting unit 7400 detects informationabout the outside of the vehicle including the vehicle control system7000. For example, the outside-vehicle information detecting unit 7400is connected with at least one of an imaging section 7410 and anoutside-vehicle information detecting section 7420. The imaging section7410 includes at least one of a time-of-flight (ToF) camera, a stereocamera, a monocular camera, an infrared camera, and other cameras. Theoutside-vehicle information detecting section 7420, for example,includes at least one of an environmental sensor for detecting currentatmospheric conditions or weather conditions and a peripheralinformation detecting sensor for detecting another vehicle, an obstacle,a pedestrian, or the like on the periphery of the vehicle including thevehicle control system 7000.

The environmental sensor, for example, may be at least one of a raindrop sensor detecting rain, a fog sensor detecting a fog, a sunshinesensor detecting a degree of sunshine, and a snow sensor detecting asnowfall. The peripheral information detecting sensor may be at leastone of an ultrasonic sensor, a radar device, and a LIDAR device (Lightdetection and Ranging device, or Laser imaging detection and rangingdevice). Each of the imaging section 7410 and the outside-vehicleinformation detecting section 7420 may be provided as an independentsensor or device, or may be provided as a device in which a plurality ofsensors or devices are integrated.

FIG. 14 depicts an example of installation positions of the imagingsection 7410 and the outside-vehicle information detecting section 7420.Imaging sections 7910, 7912, 7914, 7916, and 7918 are, for example,disposed at at least one of positions on a front nose, sideview mirrors,a rear bumper, and a back door of the vehicle 7900 and a position on anupper portion of a windshield within the interior of the vehicle. Theimaging section 7910 provided to the front nose and the imaging section7918 provided to the upper portion of the windshield within the interiorof the vehicle obtain mainly an image of the front of the vehicle 7900.The imaging sections 7912 and 7914 provided to the sideview mirrorsobtain mainly an image of the sides of the vehicle 7900. The imagingsection 7916 provided to the rear bumper or the back door obtains mainlyan image of the rear of the vehicle 7900. The imaging section 7918provided to the upper portion of the windshield within the interior ofthe vehicle is used mainly to detect a preceding vehicle, a pedestrian,an obstacle, a signal, a traffic sign, a lane, or the like.

Incidentally, FIG. 14 depicts an example of photographing ranges of therespective imaging sections 7910, 7912, 7914, and 7916. An imaging rangea represents the imaging range of the imaging section 7910 provided tothe front nose. Imaging ranges b and c respectively represent theimaging ranges of the imaging sections 7912 and 7914 provided to thesideview mirrors. An imaging range d represents the imaging range of theimaging section 7916 provided to the rear bumper or the back door. Abird’s-eye image of the vehicle 7900 as viewed from above can beobtained by superimposing image data imaged by the imaging sections7910, 7912, 7914, and 7916, for example.

Outside-vehicle information detecting sections 7920, 7922, 7924, 7926,7928, and 7930 provided to the front, rear, sides, and corners of thevehicle 7900 and the upper portion of the windshield within the interiorof the vehicle may be, for example, an ultrasonic sensor or a radardevice. The outside-vehicle information detecting sections 7920, 7926,and 7930 provided to the front nose of the vehicle 7900, the rearbumper, the back door of the vehicle 7900, and the upper portion of thewindshield within the interior of the vehicle may be a LIDAR device, forexample. These outside-vehicle information detecting sections 7920 to7930 are used mainly to detect a preceding vehicle, a pedestrian, anobstacle, or the like.

Returning to FIG. 13 , the description will be continued. Theoutside-vehicle information detecting unit 7400 makes the imagingsection 7410 image an image of the outside of the vehicle, and receivesimaged image data. In addition, the outside-vehicle informationdetecting unit 7400 receives detection information from theoutside-vehicle information detecting section 7420 connected to theoutside-vehicle information detecting unit 7400. In a case where theoutside-vehicle information detecting section 7420 is an ultrasonicsensor, a radar device, or a LIDAR device, the outside-vehicleinformation detecting unit 7400 transmits an ultrasonic wave, anelectromagnetic wave, or the like, and receives information of areceived reflected wave. On the basis of the received information, theoutside-vehicle information detecting unit 7400 may perform processingof detecting an object such as a human, a vehicle, an obstacle, a sign,a character on a road surface, or the like, or processing of detecting adistance thereto. The outside-vehicle information detecting unit 7400may perform environment recognition processing of recognizing arainfall, a fog, road surface conditions, or the like on the basis ofthe received information. The outside-vehicle information detecting unit7400 may calculate a distance to an object outside the vehicle on thebasis of the received information.

In addition, on the basis of the received image data, theoutside-vehicle information detecting unit 7400 may perform imagerecognition processing of recognizing a human, a vehicle, an obstacle, asign, a character on a road surface, or the like, or processing ofdetecting a distance thereto. The outside-vehicle information detectingunit 7400 may subject the received image data to processing such asdistortion correction, alignment, or the like, and combine the imagedata imaged by a plurality of different imaging sections 7410 togenerate a bird’s-eye image or a panoramic image. The outside-vehicleinformation detecting unit 7400 may perform viewpoint conversionprocessing using the image data imaged by the imaging section 7410including the different imaging parts.

The in-vehicle information detecting unit 7500 detects information aboutthe inside of the vehicle. The in-vehicle information detecting unit7500 is, for example, connected with a driver state detecting section7510 that detects the state of a driver. The driver state detectingsection 7510 may include a camera that images the driver, a biosensorthat detects biological information of the driver, a microphone thatcollects sound within the interior of the vehicle, or the like. Thebiosensor is, for example, disposed in a seat surface, the steeringwheel, or the like, and detects biological information of an occupantsitting in a seat or the driver holding the steering wheel. On the basisof detection information input from the driver state detecting section7510, the in-vehicle information detecting unit 7500 may calculate adegree of fatigue of the driver or a degree of concentration of thedriver, or may determine whether the driver is dozing. The in-vehicleinformation detecting unit 7500 may subject an audio signal obtained bythe collection of the sound to processing such as noise cancelingprocessing or the like.

The integrated control unit 7600 controls general operation within thevehicle control system 7000 in accordance with various kinds ofprograms. The integrated control unit 7600 is connected with an inputsection 7800. The input section 7800 is implemented by a device capableof input operation by an occupant, such, for example, as a touch panel,a button, a microphone, a switch, a lever, or the like. The integratedcontrol unit 7600 may be supplied with data obtained by voicerecognition of voice input through the microphone. The input section7800 may, for example, be a remote control device using infrared rays orother radio waves, or an external connecting device such as a mobiletelephone, a personal digital assistant (PDA), or the like that supportsoperation of the vehicle control system 7000. The input section 7800 maybe, for example, a camera. In that case, an occupant can inputinformation by gesture. Alternatively, data may be input which isobtained by detecting the movement of a wearable device that an occupantwears. Further, the input section 7800 may, for example, include aninput control circuit or the like that generates an input signal on thebasis of information input by an occupant or the like using theabove-described input section 7800, and which outputs the generatedinput signal to the integrated control unit 7600. An occupant or thelike inputs various kinds of data or gives an instruction for processingoperation to the vehicle control system 7000 by operating the inputsection 7800.

The storage section 7690 may include a read only memory (ROM) thatstores various kinds of programs executed by the microcomputer and arandom access memory (RAM) that stores various kinds of parameters,operation results, sensor values, or the like. In addition, the storagesection 7690 may be implemented by a magnetic storage device such as ahard disc drive (HDD) or the like, a semiconductor storage device, anoptical storage device, a magneto-optical storage device, or the like.

The general-purpose communication I/F 7620 is a communication I/F usedwidely, which communication I/F mediates communication with variousapparatuses present in an external environment 7750. The general-purposecommunication I/F 7620 may implement a cellular communication protocolsuch as global system for mobile communications (GSM, trademark),worldwide interoperability for microwave access (WiMAX), long termevolution (LTE)), LTE-advanced (LTE-A), or the like, or another wirelesscommunication protocol such as wireless LAN (referred to also aswireless fidelity (Wi-Fi), Bluetooth, or the like. The general-purposecommunication I/F 7620 may, for example, connect to an apparatus (forexample, an application server or a control server) present on anexternal network (for example, the Internet, a cloud network, or acompany-specific network) via a base station or an access point. Inaddition, the general-purpose communication I/F 7620 may connect to aterminal present in the vicinity of the vehicle (which terminal is, forexample, a terminal of the driver, a pedestrian, or a store, or amachine type communication (MTC) terminal) using a peer to peer (P2P)technology, for example.

The dedicated communication I/F 7630 is a communication I/F thatsupports a communication protocol developed for use in vehicles. Thededicated communication I/F 7630 may implement a standard protocol such,for example, as wireless access in vehicle environment (WAVE), which isa combination of institute of electrical and electronic engineers (IEEE)802.11p as a lower layer and IEEE 1609 as a higher layer, dedicatedshort range communications (DSRC), or a cellular communication protocol.The dedicated communication I/F 7630 typically carries out V2Xcommunication as a concept including one or more of communicationbetween a vehicle and a vehicle (Vehicle to Vehicle), communicationbetween a road and a vehicle (Vehicle to Infrastructure), communicationbetween a vehicle and a home (Vehicle to Home), and communicationbetween a pedestrian and a vehicle (Vehicle to Pedestrian).

The positioning section 7640, for example, performs positioning byreceiving a global navigation satellite system (GNSS) signal from a GNSSsatellite (for example, a GPS signal from a global positioning system(GPS) satellite), and generates positional information including thelatitude, longitude, and altitude of the vehicle. Incidentally, thepositioning section 7640 may identify a current position by exchangingsignals with a wireless access point, or may obtain the positionalinformation from a terminal such as a mobile telephone, a personalhandyphone system (PHS), or a smart phone that has a positioningfunction.

The beacon receiving section 7650, for example, receives a radio wave oran electromagnetic wave transmitted from a radio station installed on aroad or the like, and thereby obtains information about the currentposition, congestion, a closed road, a necessary time, or the like.Incidentally, the function of the beacon receiving section 7650 may beincluded in the dedicated communication I/F 7630 described above.

The in-vehicle device I/F 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 present within the vehicle. The in-vehicledevice I/F 7660 may establish wireless connection using a wirelesscommunication protocol such as wireless LAN, Bluetooth, near fieldcommunication (NFC), or wireless universal serial bus (WUSB). Inaddition, the in-vehicle device I/F 7660 may establish wired connectionby universal serial bus (USB), high-definition multimedia interface(HDMI, trademark), mobile high-definition link (MHL), or the like via aconnection terminal (and a cable if necessary) not depicted in thefigures. The in-vehicle devices 7760 may, for example, include at leastone of a mobile device and a wearable device possessed by an occupantand an information device carried into or attached to the vehicle. Thein-vehicle devices 7760 may also include a navigation device thatsearches for a path to an arbitrary destination. The in-vehicle deviceI/F 7660 exchanges control signals or data signals with these in-vehicledevices 7760.

The vehicle-mounted network I/F 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The vehicle-mounted network I/F 7680 transmits andreceives signals or the like in conformity with a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the integrated control unit 7600 controls thevehicle control system 7000 in accordance with various kinds of programson the basis of information obtained via at least one of thegeneral-purpose communication I/F 7620, the dedicated communication I/F7630, the positioning section 7640, the beacon receiving section 7650,the in-vehicle device I/F 7660, and the vehicle-mounted network I/F7680. For example, the microcomputer 7610 may calculate a control targetvalue for the driving force generating device, the steering mechanism,or the braking device on the basis of the obtained information about theinside and outside of the vehicle, and output a control command to thedriving system control unit 7100. For example, the microcomputer 7610may perform cooperative control intended to implement functions of anadvanced driver assistance system (ADAS) which functions includecollision avoidance or shock mitigation for the vehicle, followingdriving based on a following distance, vehicle speed maintainingdriving, a warning of collision of the vehicle, a warning of deviationof the vehicle from a lane, or the like. In addition, the microcomputer7610 may perform cooperative control intended for automatic driving,which makes the vehicle to travel autonomously without depending on theoperation of the driver, or the like, by controlling the driving forcegenerating device, the steering mechanism, the braking device, or thelike on the basis of the obtained information about the surroundings ofthe vehicle.

The microcomputer 7610 may generate three-dimensional distanceinformation between the vehicle and an object such as a surroundingstructure, a person, or the like, and generate local map informationincluding information about the surroundings of the current position ofthe vehicle, on the basis of information obtained via at least one ofthe general-purpose communication I/F 7620, the dedicated communicationI/F 7630, the positioning section 7640, the beacon receiving section7650, the in-vehicle device I/F 7660, and the vehicle-mounted networkI/F 7680. In addition, the microcomputer 7610 may predict danger such ascollision of the vehicle, approaching of a pedestrian or the like, anentry to a closed road, or the like on the basis of the obtainedinformation, and generate a warning signal. The warning signal may, forexample, be a signal for producing a warning sound or lighting a warninglamp.

The sound/image output section 7670 transmits an output signal of atleast one of a sound and an image to an output device capable ofvisually or auditorily notifying information to an occupant of thevehicle or the outside of the vehicle. In the example of FIG. 13 , anaudio speaker 7710, a display section 7720, and an instrument panel 7730are illustrated as the output device. The display section 7720 may, forexample, include at least one of an on-board display and a head-updisplay. The display section 7720 may have an augmented reality (AR)display function. The output device may be other than these devices, andmay be another device such as headphones, a wearable device such as aneyeglass type display worn by an occupant or the like, a projector, alamp, or the like. In a case where the output device is a displaydevice, the display device visually displays results obtained by variouskinds of processing performed by the microcomputer 7610 or informationreceived from another control unit in various forms such as text, animage, a table, a graph, or the like. In addition, in a case where theoutput device is an audio output device, the audio output deviceconverts an audio signal constituted of reproduced audio data or sounddata or the like into an analog signal, and auditorily outputs theanalog signal.

Incidentally, at least two control units connected to each other via thecommunication network 7010 in the example depicted in FIG. 13 may beintegrated into one control unit. Alternatively, each individual controlunit may include a plurality of control units. Further, the vehiclecontrol system 7000 may include another control unit not depicted in thefigures. In addition, part or the whole of the functions performed byone of the control units in the above description may be assigned toanother control unit. That is, predetermined arithmetic processing maybe performed by any of the control units as long as information istransmitted and received via the communication network 7010. Similarly,a sensor or a device connected to one of the control units may beconnected to another control unit, and a plurality of control units maymutually transmit and receive detection information via thecommunication network 7010.

It should be noted that computer programs for realizing the respectivefunctions of an image sensor 10 according to the present embodiment canbe implemented in any one of the control units etc. Also, acomputer-readable recording medium storing such computer programs can beprovided. The recording medium is, for example, a magnetic disk, anoptical disc, a magneto-optical disc, a flash memory, and the like.Also, the above computer programs may be delivered, for example, via anetwork rather than using a recording medium.

In the vehicle control system 7000 described above, the image sensor 10according to the present embodiment can be applied to the integratedcontrol unit 7600 as an application example illustrated in FIG. 13 .

Further, at least a portion of configuration elements of the imagesensor 10 may be implemented in a module (e.g., an integrated circuitmodule configured by one die) for the integrated control unit 7600illustrated in FIG. 13 . Alternatively, the image sensor 10 may beimplemented by using a plurality of control units of the vehicle controlsystem 7000 illustrated in FIG. 13 .

Note that, embodiments of the present disclosure are not limited tothose described above and can be modified in various ways withoutdeparting from the gist of the present disclosure.

The present disclosure can have the following configurations:

A solid-state image pickup apparatus including:

-   a pixel array in which a first pixel in which an OCL (On Chip Lens)    of a standard size is formed and a second pixel in which an OCL of a    size different from the standard size is formed are present mixedly;    and-   a correction section that corrects a pixel value of the first pixel    that is positioned in the vicinity of the second pixel among the    first pixels on the pixel array.

The solid-state image pickup apparatus according to (1), in which

the correction section corrects a pixel value of a third pixel that ispositioned in the vicinity of the second pixel and in which a shape ofthe OCL is deformed from an original standard size among the firstpixels on the pixel array.

The solid-state image pickup apparatus according to (2), in which

an OCL having a size larger than the standard size is formed in thesecond pixel.

The solid-state image pickup apparatus according to (2) or (3), in which

the correction section calculates a correction value that is replacedwith a pixel value of the third pixel as a correction target by using apixel value of the first pixel that is positioned in the vicinity of thethird pixel as the correction target.

The solid-state image pickup apparatus according to any one of (1) to(4), in which

the correction section decides a threshold for determining whether ornot the calculated correction value is applied.

The solid-state image pickup apparatus according to (5), in which

the correction section decides the threshold by using a parameter thatis different depending on a positional relationship between the thirdpixel as the correction target and the second pixel.

The solid-state image pickup apparatus according to (5), in which

the correction section decides the threshold by using a parameter thatis different depending on a position in an image of the third pixel asthe correction target.

The solid-state image pickup apparatus according to any one of (2) to(7), in which

the correction section determines whether or not the calculatedcorrection value is applied on the basis of a flatness in a peripheralarea of the third pixel as the correction target.

The solid-state image pickup apparatus according to any one of (1) to(8), in which

the second pixel is a phase difference detection pixel.

The solid-state image pickup apparatus according to any one of (1) to(9), further including:

an interpolation section that interpolates a pixel value correspondingto a position of the second pixel.

A method for correcting a solid-state image pickup apparatus including apixel array in which a first pixel in which an OCL (On Chip Lens) of astandard size is formed and a second pixel in which an OCL of a sizedifferent from the standard size is formed are present mixedly and acorrection section that corrects an output of the pixel array, themethod including:

a correction step of correcting, by the correction section, a pixelvalue of the first pixel that is positioned in the vicinity of thesecond pixel among the first pixels on the pixel array.

An electronic apparatus including:

-   a solid-state image pickup apparatus to be mounted thereon, in which-   the solid-state image pickup apparatus includes    -   a pixel array in which a first pixel in which an OCL (On Chip        Lens) of a standard size is formed and a second pixel in which        an OCL of a size different from the standard size is formed are        present mixedly, and    -   a correction section that corrects a pixel value of the first        pixel that is positioned in the vicinity of the second pixel        among the first pixels on the pixel array.

Reference Signs List

10 Image sensor, 11 Pixel array, 12 Phase difference detection section,13 Irregular pixel interpolation section, 14 AF control section, 15Deformed pixel correction section, 16 Defective pixel correctionsection, 17 Camera signal processing section, 21 Correction valuecalculation section, 22 Flatness determination section, 23 Thresholdcalculation section, 24 Average value calculation section, 25 Correctionsection

1. A solid-state image pickup apparatus, comprising: a pixel arrayhaving first pixels each with an OCL (On Chip Lens) of a standard sizeformed thereon and a second pixel with an OCL of a size different thanthe standard size formed thereon arranged mixedly; and a correctionsection that corrects a pixel value of a first pixel that is positionedin a vicinity of the second pixel among the first pixels on the pixelarray, wherein the second pixel is a phase difference detection pixel.2. The solid-state image pickup apparatus according to claim 1, whereinthe correction section corrects a pixel value of a third pixel that ispositioned in the vicinity of the second pixel and in which a shape ofthe OCL is deformed from an original standard size among the firstpixels on the pixel array.
 3. The solid-state image pickup apparatusaccording to claim 1, wherein an OCL having a size larger than thestandard size is formed in the second pixel.
 4. The solid-state imagepickup apparatus according to claim 2, wherein the correction sectioncalculates a correction value that is replaced with a pixel value of thethird pixel as a correction target by using a pixel value of the firstpixel that is positioned in a vicinity of the third pixel as thecorrection target.
 5. The solid-state image pickup apparatus accordingto claim 4, wherein the correction section decides a threshold fordetermining whether or not the calculated correction value is applied.6. The solid-state image pickup apparatus according to claim 5, whereinthe correction section decides the threshold by using a parameter thatis different depending on a positional relationship between the thirdpixel as the correction target and the second pixel.
 7. The solid-stateimage pickup apparatus according to claim 5, wherein the correctionsection decides the threshold by using a parameter that is differentdepending on a position in an image of the third pixel as the correctiontarget.
 8. The solid-state image pickup apparatus according to claim 4,wherein the correction section determines whether or not the calculatedcorrection value is applied on a basis of a flatness in a peripheralarea of the third pixel as the correction target.
 9. The solid-stateimage pickup apparatus according to claim 1, further comprising aninterpolation section that interpolates a pixel value corresponding to aposition of the second pixel.
 10. A method for correcting a solid-stateimage pickup apparatus including a pixel array having first pixels eachwith an OCL (On Chip Lens) of a standard size formed thereon and asecond pixel with an OCL of a size different than the standard sizeformed thereon arranged mixedly and a correction section that correctsan output of the pixel array, the method comprising: a correction stepof correcting, by the correction section, a pixel value of a first pixelthat is positioned in a vicinity of the second pixel among the firstpixels on the pixel array, wherein the second pixel is a phasedifference detection pixel.
 11. The method for correcting a solid-stateimage pickup apparatus according to claim 10, further comprising thestep of correcting, by the correction section, a pixel value of a thirdpixel that is positioned in the vicinity of the second pixel and inwhich a shape of the OCL is deformed from an original standard sizeamong the first pixels on the pixel array.
 12. The method for correctinga solid-state image pickup apparatus according to claim 11, furthercomprising the step of calculating, by the correction section, acorrection value that is replaced with a pixel value of the third pixelas a correction target by using a pixel value of the first pixel that ispositioned in a vicinity of the third pixel as the correction target.13. The method for correcting a solid-state image pickup apparatusaccording to claim 12, further comprising the step of deciding, by thecorrection section, a threshold for determining whether or not thecalculated correction value is applied.
 14. The method for correcting asolid-state image pickup apparatus according to claim 13, furthercomprising the step of deciding, by the correction section, thethreshold by using a parameter that is different depending on apositional relationship between the third pixel as the correction targetand the second pixel.
 15. The method for correcting a solid-state imagepickup apparatus according to claim 13, further comprising the step ofdeciding, by the correction section, the threshold by using a parameterthat is different depending on a position in an image of the third pixelas the correction target.
 16. The method for correcting a solid-stateimage pickup apparatus according to claim 12, further comprising thestep of determining, by the correction section, whether or not thecalculated correction value is applied on a basis of a flatness in aperipheral area of the third pixel as the correction target.
 17. Themethod for correcting a solid-state image pickup apparatus according toclaim 10, further comprising the step of interpolating, by aninterpolation section, a pixel value corresponding to a position of thesecond pixel.
 18. An electronic apparatus comprising: a solid-stateimage pickup apparatus to be mounted thereon, wherein the solid-stateimage pickup apparatus includes: a pixel array having first pixels eachwith an OCL (On Chip Lens) of a standard size formed thereon and asecond pixel with an OCL of a size different than the standard sizeformed thereon arranged mixedly, and a correction section that correctsa pixel value of a first pixel that is positioned in a vicinity of thesecond pixel among the first pixels on the pixel array, wherein thesecond pixel is a phase difference detection pixel.
 19. The electronicapparatus according to claim 18, wherein the correction section correctsa pixel value of a third pixel that is positioned in the vicinity of thesecond pixel and in which a shape of the OCL is deformed from anoriginal standard size among the first pixels on the pixel array. 20.The electronic apparatus according to claim 18, wherein an OCL having asize larger than the standard size is formed in the second pixel.